On - Line Learning from Restricted Training Sets

{ We analyse the dynamics of on-line learning in multilayer neural networks where training examples are sampled with repetition and where the number of examples scales with the number of network weights. The analysis is based on monitoring a set of macroscopic variables from which the training and generalisation errors can be calculated. A closed set of dynamical equations is derived using the dynamical replica method and is solved numerically. The theoretical results are consistent with those obtained by computer simulations. Layered neural networks are powerful nonlinear information processing systems, capable of implementing arbitrary continuous and discrete input-output maps to any desired accuracy 1], given a suucient number of hidden nodes and a suuciently large example set. They have been employed successfully in a variety of regression and classiication tasks, and have been studied using a wide range of methods (for a review see 2]). On-line learning refers to the iterative modiication of the network parameters according to a predetermined training rule, following successive presentations of single training examples, each representing a speciic input vector and the corresponding output. On-line learning is one of the leading techniques in training large neural networks, especially via gradient descent on a diierentiable error measure. Considerable progress has been made recently in analysing the dynamics of supervised on-line learning in layered neural networks via methods of statistical physics (reviews can be found in 3] and 4]). Most of the analyses (e.g. 5, 6, 7]) have concentrated on the case of uncorrelated innnite training sets, where training examples are sampled without repetition and in which there is no correlation between the network parameters and the examples presented at each training step. However, a more realistic scenario is that where the number of training examples scales with the number of free parameters, and are sampled with repetition. This gives rise to correlations between the network parameters and the training examples, which clearly aaect the learning process. One of the most signiicant aspects of having a xed Typeset using EURO-L A T E X